(N 



When Do Phylogenetic Mixture Models Mimic Other 

Phylogenetic Models? 

Elizabeth S. Allman^, John A. Rhodes^*, Seth Sulhvant^ 
^Department of Mathematics and Statistics, University of Alaska Fairbanks, Box 756660, 

Fairbanks, AK, 99775 
'^Department of Mathematics, North Carolina State University, Box 8205 Raleigh, NC 

27695 



O ■ * To whom correspondence should be addressed; 

r\ • E-mail: j.rhodes@alaska.edu 

Abstract— Phylogenetic mixture models, in which the sites in sequences undergo differ- 
ent substitution processes along the same or different trees, allow the description of heteroge- 
PLh ■ neous evolutionary processes. As data sets consisting of longer sequences become available. 



it is important to understand such models, for both theoretical insights and use in statistical 



O 

Q^! analyses. Some recent articles have highlighted disturbing "mimicking" behavior in which 



a distribution from a mixture model is identical to one arising on a different tree or trees. 

>. 

\0 . Other works have indicated such problems are unlikely to occur in practice, as they require 

a^■ 

^ ! very special parameter choices. 

g : After surveying some of these works on mixture models, we give several new results. 

^ " In general, if the number of mixture components is not too large, and we disallow zero or 

. infinite branch lengths, then mimicking does not occur. On the other hand, if the mixture 

X. 

H . model is locally over-parameterized, it is possible for a phylogenetic mixture model to mimic 

distributions of another tree model. Though theoretical questions remain, these sorts of 
results can serve as a guide to when the use of mixture models in either ML or Bayesian 
frameworks is likely to lead to statistically consistent inference. 

Keywords: Phylogenetic mixture models, parameter identifiability, heterogeneous sequence 
evolution 
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Model-based phylogenetic inference from sequence data requires compromises between sim- 
plicity and biological realism. Typical current modeling assumptions include that all sites 
evolve on a single tree, according to the same substitution process, often with a simple F- 
distributed scaling of rates across the sites. While one can easily formulate models allowing 
more complexity, the additional parameters this introduces can be problematic. Not only 
is software likely to require longer run-times, but one also risks 'overfitting' of finite data 
sets and thus interpreting stochastic variation as meaningful signal. In more extreme cases, 
over-parameteriztion can lead to loss of identifiability of some parameter of interest, and 
thus the loss of statistical consistency of inference as well. 

Nonetheless, it is not hard to conceive of data sets for which the modeling assumptions 
underlying a routine analysis are strongly violated. For instance, different parts of a single 
gene sequence might undergo rather different substitution processes, perhaps due to different 
substructures of the protein they encode. Alternatively, lateral transfer of genetic material 
may have resulted in sequences that are amalgams of those evolving on different trees. 
Analyzing such data under a standard model simply assumes that neither of these has 
occurred, and so uses a misspecified model. While one would hope there would be some 
indication of this as the analysis is conducted — perhaps by a poor likelihood score or poor 
convergence of a Bayesian MCMC run — there is no guarantee of an obvious sign of a 
problem. 

An alternative is to consider mixture models, which explicitly allow for such heterogeneity 
in the data. Mixtures consider several classes of sites which might each evolve according to a 
distinct process, either on the same tree (a single-tree mixture model), or on possibly different 
trees (a multitree mixture model). In both cases the use of a mixture model differs from a 
partitioned analysis of data, in which the researcher imposes a partitioning of the sites into 
classes, each of which must evolve according to a single standard model. For a mixture 
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model, there is no a priori partitioning; the proportion of sites in each class is a parameter 
of the model, and is thus to be inferred. 

The single-tree GTR+r(+I) model is a familiar, but highly restricte d type of mixture, 



with 



Chai and Housworth 



ew parameters, that is commonly used in data analysis. Only recently!; 
(j201ll ) completed a rigorous proof that the parameters of this model, including the tree topol- 
ogy, are identifiable from its probability distributions in most cases, and thus that it gives 
consistent inference und er maximum likelihood. However, the special case of the F81+r+I 



submodel remains open (jStee 



20091). 



On the other hand, a single-tree rate-variation model in which the rate distribution was 
allowed to be arbitrary was one of the earliest mixture models seen to be problematic, as 



every tree can produce the same distribution of site patterns ( 



Steel et a. 



Tuffley and Steel! (119971 ) provides another 



1994! ). The no- 



common-mechanism (NCM) model introduced by 
example of a mixture in which distributions do not identify trees. However, these models are 
rather unusual, in that the number of their parameters grows with sequence length. This 
extreme over-parameterizaton is well understood, as is the implication that these mod els do 



not le ad to statistically consistent inference under a maximum likelihood framework. (jStee 
(|201l! ) offers a more complete and subtle discussion of NCM models and inference.) Of 
course these models were introduced to elucidate theoretical points, and were not intended 
for data analysis. 

Much recent work on mixture models has focused on those with a finite (though perhaps 
large) number of mixture components, allowing more heterogeneity among the classes than 
the simple scaling of the rate variation models. Several papers have shown that inference from 
data generated b y a mixture process can be poor if th e analysis is based on a misspecifie d non- 



mixture model ( iKolaczkowski and Thornton . 



120041 : 



Mossel and Vigoda . 



2005 



20061 ). The 



examples in these works indicate that we may be misled if we ignore the 



such heterogeneity. This point is further underscored by 



Matsen and Steel! fl2007h . who 



p ossibi lity of 



discuss why analysis with a misspecified non-mixture can lead to erroneous inference in some 
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specific circumstances. As there is no general reason why one should expect good inference 
with a misspecified model, to our mind these works primarily indicate the importance of 
further study of mixture models, so that they may be applied intelligently when substantial 
heterogeneity is possibly present. 

However, several works have indicted that models with a finite number of mixture classes 
may ha ve theoretical shortcomings as w ell. Working with no restriction on the number of 



classes. 



Stefankovic and Vigodal (l2007a 



jb|) emphasize that unless a model is special enough 
that there are linear inequalities (which they call linear tests) distinguishing between un- 
mixed distributions arising on different trees, then t here will be cases in w hich trees can not 



be identified from single-tree mixture distributions, 
particularly for the Cavender-Farris-Neyman (CFN 



Matsen et al. 



( I2OO8I ) explore this more 



ing models with a smal. 



number of mixture classes. 



2-state symmetric model. Co nsider- 



Stefankovic and Vigodal (l2007a 



b|) and 



Matsen and Steell (120071 ) give explicit examples of parameter choices in certain 2-class CFN 



single-tree mixture models that lead to exactly the same unmixed probability distributions 
s on a different tr e e. Su ch non-identifiability of the tree, or 'mimicking' 



as standard mode 

as it is termed by iMatsen and Steell (l2007l ) , means that the true tree parameter cannot be 
determined even from an exact theoretical probability distribution, much less from data 
sampled from it. If the problems highlighted in these works were widespread, then we would 
be severely limited in our ability to detect heterogeneous process. Moreover, heterogeneous 
processes on one tree might routinely mislead us into thinking data arose on a different 
tree. We have encountered researchers who, not surprisingly, found these possibilities quite 
alarming. 

While there is no doubt that certain mixtures are problematic, whether this is really of 
great practical concern is in fact not at all clear from the results mentioned so far. Thought- 
ful use of mixtu re models for data analysis has seemed to perforin well f or a number o: 



2008 



resea r ch groups (|Ronauist and Huelsenbeckl. 



Wang et al. 



2008 



2003 



Evans and Sullivan . 



Pagel and Meade 



2004 



2005 



Le et al. 



2OI2I ). While publication bias against failed 
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analyses could be responsible for a lack of reports of difficulties with mixture models in the 
literature, we also h ave not heard of such prob l ems through our pro f essional interactions. 



Several papers (lAllman and Rhodes . 



2006 



AUman et al. 



2011 



Rhodes and Sullivant 



2OI2I ) have given a strong theoretical indication that problematic mixtures, for which trees 
are non-identifiable, are quite rare. Using algebraic techniques building on the idea of phy- 
logenetic invariants, these works show in a variety of contexts that mixture distributions 
cannot mimic distributions arising on other trees, for generic choices of numerical param- 
eters. 'Generic' here has a precise meaning that informally can be expressed as "if the 
model parameters are chosen at random, and thus do not have any special values or rela- 
tionships among themselves." More formally, the set of exceptional parameters leading to 
non-identifiability is of strictly smaller dimension than the full parameter space. Thus if 
the true parameters were chosen b y throwing a dart at the par ameter space, they would be 



sure to lie off that exceptional set. 



Rhodes and Sullivant 



( 120121 ) give an upper bound on the 



number of classes that, for a quite general model, ensures generic identifiability of the trees 
in all single-tree and in many multitree mixtures. This bound is exponential in the number 
of taxa, and likely to be larger than the number of classes one would actually use in data 
analysis. 

While these positive theoretical results indicate one should seldom encounter problems 
with the judicious use of a mixture model in data analysis, one may still worry about the 
possible exceptions. The exceptional cases are generally not explicitly characterized in these 
papers, and the arguments used to establish that they form a set of lower dimension are rather 
technical. The intuition of the authors is that the potential exceptional set one could extract 
from these works are likely to be much larger than the true exceptional set, as an artifact of 
the techniques of proof. Moreover, experience with other types of statistical models outside 
of phylogenetics {e.g., hidden Markov models, Bayesian networks) with similar exceptional 
sets of non-identifiability has shown they can still be quite useful, and are generally not 
problematic for data analysis. 
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In this work we first give mathematical justification — with no cryptic assumptions of 
genericity of parameters — that a hmited amount of heterogeneity in a single-tree mixture 
cannot mimic evolution on a different tree in most relevant circumstances. We also show how 
examples of non-identifiability of trees due to mixture processes can arise from a readily un- 
de rstood issue of local over - yarame terizati on. This explains t he 2-c lass mimicking examples 



of 



Stefankovic and Vigodal (l2007al Jbl) and 



Matsen and Steell (120071 ). which are constructed 



for models whose parameter space is of larger dimension than the distribution space for a 
4-taxon tree. However, this is not the setting in which most data analysis is likely to take 
place. For 4-state models encompassing those such as the general time-reversible (GTR) 
which are in common use, we show even 3-component mixtures cannot mimic non-mixtures. 
While our positive identifiability result s do not allow as many mixt ure components as the 
ones obtained for generic parameters by 



Rhodes and Sulhvant 



(|2012[ ). by excluding the pos- 



sibility of exceptions they are, in some sense, more complete. Finally, for certain group-based 
models (Jukes-Cantor and Kimura 2 parameter), for which linear tests exist, we also obtain 
results indicating that if mimicking does occur for multitree mixtures, then it is not entirely 
misleading. In the case of fully-resolved trees, any mimicking distribution can only agree 
with a distribution coming from one of the topological trees appearing in the mixture. 

The description of these results here provides only an outline, as precise statements 
require a thorough specification of the basic model. We therefore formulate the general 
continuous-time model, and mixtures arising from it, in the next section. Then in a subse- 
quent section a number of theorems formally encapsulate the claims above. 

Our results concern only the theoretical limits of what one might be able to infer from a 
data set; we do not study any issues relating to the performance of mixture models for 
inference from finite data sets. Our hope is to further dispel concerns that the use of 
phylogenetic mixture models is inherently problematic, or that inhomogeneous processes 
are likely to lead to inherently undetectable mistakes in most settings in which inference is 
performed. As is the case for any statistical model, phylogenetic mixture models must be 
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applied thoughtfully, and whether they are useful or not will depend on the data. 

The mathematical tools we use to obtain our results involve the polynomial equalities 
called phylognetic invariants, which have been extensively studied for both the group-based 
models and the general Markov model, and mixtures built from them. However, we supple- 
ment these with some polynomial inequalities . While the potential usefulness of inequalities 



was made clear even in the seminal paper of 



Cavender and Felsensteiru ( 119871 ) which intro 



duced invariants, their study unfortunately remains much less developed than the study of 
invariants. Though a deeper understanding of inequalities for both unmixed and mixture 
models would be highly desirable, here we make do with a few ad hoc ones. 

Phylogenetic Mixture Models 

In this section, we describe the class of phylogenetic models that we study. Our definition of 
an unmixed phylogenetic i nodel is broad, encompassing mo st standard phylogen e tic rn odels. 



including thos e stud ied by 



Matsen et al. 



Stefankovic and Vigodal ( l2007aU bl) 



Matsen and Steell (120071), and 



( I2OO8I ). Informally, we consider continuous-time models, but do not require 



time-reversibility or stationarity, and allow the substitution process to change at a finite set 
of points on t he tree. Such relaxations of the usual modeling assumptions have appeare d in 



several works (lYang and Roberts 



1995 



Galtier and Gouy 



1998 



Yap and Speed 



20051 ). 



We assume that the random variables modeling characters have k > 2 states, the most 
important values being k = 4 (DNA models), k = 2 (purine/pyrimidine models), and k = 20 
(protein models). 

By a rate matrix for a state substitution process we mean a k x k matrix with nonnegative 
off-diagonal entries, whose row sums are all zero. (To fix a scaling, one may also impose 
some normalization convention.) Such a rate matrix Q = (qij) generates a continuous-time 
K-state Markov chain. Associated with Q is a directed graph, Gq, on nodes {1, 2, . . . , k} 
which has an edge z — j- j if and only if qij ^ 0. The process defined by Q is irreducible if. 
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and only if, Gq is strongly connected, that is, there is a directed path from node i to node 
j for all Irreducibility guarantees that for all t > the discrete-time Markov transition 
matrix exp(Qt) has strictly positive entries. Of course exp((5t) is the identity matrix when 
t = 0, and so has zero entries. 

Consider an unrooted, combinatorial, phylogenetic tree, T, in which we allow polytomies. 
Then by the general continuous-time model on T, we mean the following: First, possibly 
introduce a finite number of degree 2 nodes (in order to model a root, and points where 
the state substitution process changes) along any of the edges of T to obtain T' . Then 
choose some node to serve as a root of T', and make any assignment of a strictly positive 
K-state distribution tt at the root. Irreducible rate matrices Qi and edge lengths tj G M>o 
are assigned to each edge i of T' . This notion is more general than is often used in most 
practical data analysis, since 1) tt need not be the stationary distribution of any Qi, and 2) 
the Qi may be different for each edge; we do not assume a common process across the tree. 
We at times restrict to considering only irreducible rate matrices of a certain form {e.g., 
Jukes-Cantor, or GTR) and specialized tt, in order to draw conclusions about submodels. 

If numerical model parameters are specified as above, then the Markov transition matrix 
on edge i of T' is Mj = exp((5iti). If T" denotes the tree obtained from T' by suppressing 
non-root nodes of degree 2, and edges z, -i + 1, . . . , z + r of T' become a single edge of T", 
then one defines a Markov matrix on that edge of T" as the product MjMj+i ■ ■ ■ Mj^^. Then 
one can compute the probability distribution arising from the numerical parameters on T', 
by using the root distribution and edge transition matrices on T" in the usual way. From 
the assumption of irreducibility of rate matrices we immediately obtain the following. 

Lemma 1. Consider any choice of general continuous-time parameters on a phylogenetic 
tree T. Then the Markov transition matrices associated to the edges of T' and T" are each 
either the identity matrix, or a nonsingular matrix with strictly positive entries. 

By A^i- we denote the set of all probability distributions arising on T for all choices 
of general continuous-time parameters. Later in this paper, we use the same notation for 

8 



a submodel obtained by restricting parameters to a specific form. If no sucli restriction is 
made, we refer to A^r as the general continuous-time model on T. 

Let Ai^ C JUj. denote the subset of distributions obtained by requiring that no internal 
branch lengths are zero, that is all tj G ]R>o except possibly for pendent edges. This is the 
open phylogenetic model. Since we allow trees to have polytomies, any distribution in Mt 
is contained in the open model for a possibly different tree; one merely contracts all internal 
edges of T which were assigned branch length zero, thus introducing new polytomies. 

If T = {Ti, . . . , Tr} is a multiset of trees, then the mixture model 

Mr = * ■ ■ ■ * M-Tr 
is the set of probability distributions of the form 

SiPi + S2P2 H h SrPr, 

where Pi G M.Ti is a probability distribution arising on Tj, and the Sj > are weighting 
parameters with si + S2 + ■ ■ • + = 1. The open mixture model 

is defined similarly, with pi G M-Ti- Note that in the open mixture model we allow mixing 
parameters to be in the closed probability simplex Aj. = {s | Sj > 0, ^ Sj = 1}. If all mixing 
parameters are required to be strictly positive, we denote the set of distributions by M.^^, 
since we are then working with the open probability simplex = {s | > 0, ^ = 1}. 

Results 

Single-tree Mixture Models 



Matsen and Steell (120071 ) and IStefankovic and Vigodal (l2007bl ) showed that under the CFN 
model it is possible for a 2-class mixture on a single tree (that is, T = {T, T}) to produce 
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distributions matching those of an unmixed model on a different tree. iMatsen et al.l (12008[ ) 
showed that this is possible if, and only if, the trees involved differ by application of a single 
NNI move. 

Our main result in this setting shows that these possibilities are essentially a "fluke of 
low dimensions." In a subsequent section a further analysis will show that this mimicking is 
a consequence of local over-parametrization, arising essentially because the CFN model is a 
2-state model. 

Theorem 2. Consider the K-state general continuous-time phylogenetic model. Let T = 
{Ti,Ti, . . . ,Ti} be the multiset consisting of n — 1 copies of tree Ti, and S = {T2}. Then 
Aij- n Ai^ = unless Ti is a refinement 0/T2. 



This result was a 
more general setting ( 



ready known to ho. 



AUman and Rhodes 



d for generic choices of parameters in a slightly 



20061 ). so the contribution here is to remove the 



generic assumption. Note that for the important case of k = 4, corresponding to DNA 
models, this implies that we cannot have a two or three class mixture mimic the distribution 
on a single tree unless w e allow zero length bran ches i n the mixture compone n ts. Th is 



indicates the examples of 



Matsen and Steel! (120071 ) and 



Stefankovic and Vigodal (12007^ 



cannot be generalized to 4-state models, without passing to at least a 4-class mixture. 



Local Over-parametrization 



Note that the examples of iMatsen and Steel! (!2007l ) and !Stefankovic and Vigodal (!2007bl ) are 
allowed by Theorem [21 since they are constructed for a model with k = 2 and T a 2-element 
multiset. To see why the existence of such examples should not be too surprising, it is 
helpful to first consider a 4-leaf tree T and perform a parameter count for the CFN model. 
A 2-class single-tree mixture on T can be specified by 11 numerical parameters: for each 
class there are 5 Markov transition matrices with 1 free parameter each, and 1 additional 
mixing parameter. However any 4-taxon CFN mixture distribution on any 4-taxon tree lies 
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in a certain 7-dimensional space, due to the symmetry of the model. Although this does 
not prove every distribution with such symmetry must arise from this 2-class mixture, the 
excess of parameters suggests that it is hkely that many do. As a result, one suspects at least 
some non-mixture distributions on different trees are likely to be mimicked by this 2-class 
mixture. This suspicion is then confirmed by explicit examples. 

When a tree has many more leaves, however, a similar parameter count for the 2-class 
CFN mixture can fail to indicate potential problems, since the number of parameters grows 
linearly with the number of leaves, while the dimension of the distribution space grows 
exponentially. However, we show below how one can extend mimicking examples on small 
trees to larger trees, thus creating what might at first appear to be more unexpected instances 
of mimicking. We refer to such examples, where mimicking is produced first on a small tree 
by allowing an excessive number of mixture components, and then extended to larger trees, 
as arising from local over-parameterization. This notion can be used to produce many new 
examples of the mimicking phenomenon, on single- or multitree mixtures. 

We distinguish here between three types of mimicking, of different degrees of severity. 
For notational convenience we use to denote any of the models A^r?-^-/-' '-'^ M.j-'^- 

Definition 3. A mixture model Ai^ weakly mimics distributions in Aig if A^^fl Ai^ ^ 0. 
A mixture model Ai^ strongly mimics distributions in A^J if dimA^^fl A^J = dim AIJ. A 
mixture model Ai^ completely mimics distributions in AIJ if Aig C Ai^. 

Thus weak mimicking requires only a single instance of probability distributions arising 
on S and T matching, strong mimicking requires a neighborhood of distributions arising on 
S to be matched by ones arising on T, and complete mimicking requires every distribution 
arising on S to be matched by one arising on T. 

Let T be a tree on the leaf set X. For each x G X, let be a new, non-empty set of 
leaves, and let be a tree with root x and leaf set A^. A set of such trees B = {B^ : x G X} 
is called a set of fusion ends for X. The fusion tree T^, with leaf set Uxex^x, is obtained 
from T and B by identifying each leaf x of T with the root x on B^. (See Figured] for an 

11 




Figure 1: The fusion tree is constructed from a tree T with leaf set X = {a, b, c, d} and a 
set B = {Ba, Bf,, Be, B^} of fusion ends for X. The construction using B could be applied to 
any of the quartet trees with leaf set X, yielding fusion trees differing by an NNI move from 
the shown here. This process underlies the extension of mimicking examples on small 
trees to larger ones. 

example.) 

If T is a collection of trees each with leaf set X, and B = {B^ : x G X} is a set of 
fusion ends for X, let be the multiset = {T^ : T G T} of fusion trees. The following 
propositions allow us to pass mimicking properties from small trees to large trees. 

Proposition 4. Suppose for a taxon set X that Ai'j weakly mimics M.%, and that B is a 
set of fusion ends. Then M.*^b weakly mimics Ai*gB- 

Proof. Any distribution q G fl A^^ arises from parameters on the trees in T, as well as 
from parameters on the trees in S. Extend these parameters to the trees in and by 
choosing a length and rate matrix for each edge of each tree in B, and using these choices 
for the resulting edges in the individual fusion trees in and iS^. Using the same mixing 
parameters as led to g, these parameters give rise to a distribution G A^^g fl A^^b- n 

Proposition 5. Let S = {T}, be a single tree, and suppose that Al^ strongly mimics (or 
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completely mimics) M^- Then for any set B of fusion ends for X, Ai*^B strongly mimics 
(or completely mimics) Ai*gB- 

Proof. This follows from the same argument as for Proposition HI with the additional ob- 
servation that the parameters assigned to edges in the fusion ends can be varied arbitrarily. 
Since S consists of a single tree, this will give a give a full dimensional set of distributions 
in Ai*gB which are mimicked by distributions in Ai*^B- If A^r completely mimics A^^, note 
that every distribution in arises from our construction so that A^^g completely mimics 
M*sB. □ 

These propositions allow the construction of explicit examples of mimicking behavior on 
large trees from those found on small trees. A typical result of this type, using quartet trees 
as a basis, is: 

Theorem 6. Let T = {T12134, . . . , T12134}, with r copies of the quartet tree T12134, and 
S = {Ti3|24, . . . , T13124}, with s copies of the quartet tree T13124. Let T and T' he trees 
with n > 4 leaves that differ by an NNI move, T' = {T, . . . ,T}, with r copies of T, and 
S' = {T', . . . , T'}, with s copies ofT'. 

//Al^ weakly mimics Al^, then Ai^, weakly mimics AIJ/. Furthermore, if Ai^ strongly 
(or completely) mimics AIJ and s = 1, then Al^, strongly (or completely) mimics AIJ/ 

Proof. Two trees differ by an NNI move if and only if and only if they are obtained from 
applying fusions to two differing quartet trees. Hence, we can apply Propositions H] and 
El □ 

In particular. Theorem [6l implies that if quartets give mimicking behavior, then we 
will h ave mimicking beha vior on trees of arbitrary size. (Note conversely that Theorem 



31 of 



Matsen et al. 



(120081 ) shows that the only way 2-class single-tree CFN mixtures can 
mimic CFN non-mixtures on large trees is through such a process applied to quartet over- 
parameterization. ) 
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Consider now the general continuous-time model on a 4-leaf tree. With 5 edges, a dis- 
tribution is specified by ~ numerical parameters. Since there are no linear tests for this 
model, and the probability distribution lies in a space of dimension — 1, we expect that 
a mixture of more than ^ k^/S components will include an open subset of the probability 
simplex. Hence such a model is likely to display mimicking behavior. Thus some sort of 
mimicking seems unavoidable for even moderately sized mixtures. To illustrate, with DNA 
sequences and k = 4, an unmixed model is specified by 63 parameters, so the 4-class mix- 
ture model has enough parameters that it is likely to include a full-dimensional subset and 
produce mimicking. 

Note that mimicking of the sort produced by local over-parameterization as above need 
not be limited to that arising from quartet trees. With enough mixture components, for some 
models it may be possible for a mixture on a relatively small tree to mimic a distribution 
from another tree, differing by more than a single NNI move. This mimicking would again 
extend to larger trees, using the fusion process of Propositions |4] and [51 

From a practical perspective, however, mimicking through local over-parameterization 
seems unlikely to be much of an issue in most data analyses, since the mixture parameters 
leading to it require that the mixed processes differ only on a small part of the tree, and 
are identical elsewhere. Researchers studying biological situations in which this might be 
plausible should, however, be aware of the possibility. 

Finally, we emphasize that we have not shown that local over-parameterization is the only 
possible source of mimicking. It would be q uite interesting to hav e examples of mimicking of 



other sorts, or extensions of Theorem 31 of 
mixture components. 



Matsen et al. 



(120081 ) to other models and more 
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Models with Linear Tests 



An early motivation for the study of linear invariants for phylogenetic models was that they 
are also invariants for mixture models on a single tree, and thus offered hope for determin- 



ing tree topo 



performance (iHuelsenbeck 



ogies even under h eterogeneous processes across sites. While poor practical 



19951 ) even in the unmixed case led to their abandonment as an 



inference tool, they remain useful for theoretical purposes. However, among the commonly- 
studied phylogenetic models, the Jukes-Cantor (JC) and Kimura 2-parameter (K2P) models 
are the only ones which possess ph ylogenetically-informative linear invariants. 



Stefankovic and Vigodal (j2007al ) used these linear invariants and the fact that they can 
be used to give linear tests, to show that if S and T are multisets each consisting of a single 
repeated n-leaf binary (fully-resolved) tree, and these trees are different, then Ai^CiAi^ = 0, 
regardless of the number of mixture components. We explore the extent to which these results 
can be extended to nonidentical tree mixtures for the JC and K2P models. 

Theorem 7. Consider the Jukes-Cantor and Kimura 2-parameter models. Let S = {Ti, . . . , Ti} 
be a multiset of identical trees on X, and T an arbitrary multiset of trees on X . If Ais H 
Ai^^ 7^ 0> then for every four element subset K (1 X , and for all T G T, either T\x is an 
unresolved (star) tree, orT\K = Ti\k- Thus all trees in T have Ti as a binary resolution. 
Furthermore, if all trees T eT are binary, and Ti ^ T then Aig fl Ai^ = 0. 

Informally, the last statement of this theorem states that arbitrary multitree phylogenetic 
mixtures on fully-resolved trees cannot mimic mixtures on a single tree, unless that tree ap- 
pears in some component of the mixture. Thus if one erroneously assumed such a mimicking 
distribution was from a single tree mixture, the single tree one would recover would in fact 
reflect the truth for at least one mixture component. 

In the case that S = {Ti}, so Ais is not a mixture but rather a standard model, for the JC 



and K 2P m odels this again rules out any m imicking examples of the sort 



Matsen and Steel 



fl2007h and 



Stefankovic and Vigodal (l2007bl ) give for CFN, unless one allows zero length 
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branches. This clearly indicates the special nature that any such exceptional cases must 
have. 

Our final theorem provides a construction of the special case of mimicking allowed by 
Theorem [TJ for the Jukes-Cantor model, where T contains nonbinary trees that are degen- 
erations of the tree T. 

Theorem 8. Let T be a tree with internal vertex v which is adjacent to three other vertices 
Ui,U2,U3. For i = 1,2, let Ti he the tree obtained from T by contracting the edge UiV. Let 
S = {T} and T = {Ti,T2}. Then, under the Jukes-Cantor model completely mimics 
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Appendix: Mathematical Arguments 

To prove Theorem |2] we first handle the special case of 4-leaf trees. We need the following 
definition. 

Definition 9. If P is a probability distribution for a K-state phylogenetic model on a n- 
taxon tree, we view it as an n-dimensional k x k x ■ ■ ■ x k, tensor, or array, of probabilities, 
P = {Piii2...i„) , where the index ii refers to the state at leaf I. Then given any bipartition 
of the leaves into non-empty subsets {1,2, ...,n} = AL\ B, the A\B flattening of P is 
the k'^' X ftl^l matrix FlatA|B with the same entries as P but with rows indexed by state 
assignments to leaves in A, and columns indexed by state assignments to leaves in B. 

Lemma 10. Consider 4-leaf trees Ti with split 12|34, andT2 either the tree with split 13|24 
or the star tree. Then the statement of Theorem\^ holds. That is, Aij- fl Ai^ = unless T2 
is the star tree. 
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Proof. Let p denote a probability distribution p G A^r? which we consider as a 4- dimensional 



tensor. Consider th e {1, 2 j 1 1 3, 4j flattening 



Allman and RhodesI (120061 ) or 



'lati2|34(p), which is a x matrix. From 



Eriksson! (120051 ) it is known that if p G Aij- then the rank of 



Flati2|34(p) is at most k(k — 1). 

On the other hand, if T2 = 13|24 and q G Aig = -Mjp^, then the matrix Flati2|34(q') has 
a factorization as 

Flati2|34(g) = (Ml ® M2)diag(Ar)(M3 M4) (1) 

where Mj, 1 < « < 4 are the transition matrices associated with the leaf edges in the tree, 
and = diag(7r)M5 where M5 is the transition matrix associated to the internal edge and 
we have assumed the tree root is at one end of that edge. Here diag(A^) denotes 
diagonal matrix constructed with the entries of on its diagonal in an appropriate order. 
By Lemma in all transitions matrices for the model A^T2 are nonsingular. Thus the k"^ x 
matrices Mi M2 and M3 ^ M4 are nonsingular. Also by Lemma [H for the open model 
A^-^^, the matrix diag(A^) is nonsingular since all the entries of tt and M5 are nonzero. Thus 
if g G A^T2' Fl&ti2|34(Q') has rank k^. 

If T2 is the star tree, then formula ([I]) still holds if one sets M5 = /. In this case the 
matrix diag(A^) is singular, and the rank of Flat 12134(g) is k. 

These conditions on the rank of Flati2|34(g) now imply the desired conclusion. □ 

Proof of Theorem\^ If Ti is a reflnement of T2, then one checks that M.^ C M-r-, by choosing 
the mixing weights as a standard unit vector, and setting edge lengths equal to zero on the 
edges appearing in Ti but not T2. 

So assume that Ti is not a reflnement of T2, yet Aij- fl Ai'g is non-empty. We may also 
assume that Ti is a binary tree, by passing to a reflnement, as this only enlarges the mixture 
model. There exists a subset K of four taxa such that the induced quartet trees Ti\k and 
T2\k are different. Marginalizing to K, since {M.-y)\j^ = J\A(^'j-\k) and {Ais)\K = A1(5|k)' 
have that A4(^r\K) ^ -^^sIk) non-empty. 

Now, by Lemma [1] the transition matrices that arise in the resulting quartet trees will 
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be products of nonsingular matrices that either are the identity, or have all positive entries. 
Thus each quartet tree transition matrix is nonsingular and can have zero entries if, and 
only if, it is the product of identity matrices. We now apply Lemma [10] to deduce that all 
the edge lengths along the internal edge of T2IX niust be zero. But this contradicts the fact 
that we were working with the open model A^^. □ 

To prove Theorem [TJ we recall a number of results about the JC and K2P models, 
including their descriptions in Fourier coordinates, and properties of linear invariants/tests 
for these models. 

The JC, K2P (and K3P) models are group-based models, with a special structure gov- 
erned by the finite abelian group G = Z2 x Z2. We associate nucleotides with elements of 
this group via 

A=(0,0), C = (0,1), G=(1,0), T=(l,l). 



The discrete Fourier transform (also called Hadamard conjugation in this context) (IHendy 



1989 



Evans and Speed 



I993I ) is an invertible linear transformation that simplifies the pa- 
rameterization of a group-based model. In Fourier coordinates, qgi...g„, the parametrization is 
described as follows: To each of the tree T's splits A\B we associate a collection of parameters 
where g E G. Then 

I otherwise. 

Proposition 11. Suppose that a transition matrix has the form exp{Qt) where Q is a rate 
matrix for a Z2 x Z2 group-based model, t > 0, and Q defines an irreducible Markov chain. 
Then the Fourier parameters satisfy the constraints: 



A\B 



A\B ^ A\B A\B 



A\B ^ A\B A\B 



A\B ^ A\B A\B 
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with , a^''^, a^''^ G (0, 1). When t = 0, all parameters equal 1. 



Additionally, under the K2P model, a^^ = a^'^, and under the JC model a 



A\B 



A\B 

c 



A\B 



A\B 



Proof. Let Q be a rate matrix of K3P format and H the associated 4x4 Hadamard matrix, 
that is, for some a, /3, 7 > 0, 5 = —a — /3 — 7, 



Q 



6 a (3 'J 

a 6 'J (3 

(3 ■y 6 a 

'J f3 a 6 



and H 



1 1 
1 -1 
1 1 
1 -1 



1 1 

1 -1 

-1 -1 

-1 1 



The Fourier coordinates for this model consist of the eigenvalues of the matrix exp(Qt). 
The matrix H consists of the eigenvectors of the matrix Q, and hence of exp{Qt). We 
compute that H~^QH is the diagonal matrix diag(0, — 2(a + 7), — 2(/3 + 7), — 2(a + /?)). 
From this we deduce that the Fourier coordinates for this model are then 

a^'^ = 1, a^'^ = exp(— 2^(0; + 7)), a^^ = exp(— 2t(/3 + 7)), = exp(— 2t(a + /?)). 

Since Q gives an irreducible Markov chain, at most one of a, /3, and 7 can be zero, which 
implies that all of a^'^, a^'^, a^'^ < 1 when t > 0. Furthermore, we see that the claimed 
inequalities hold, e.g., 

a^l^ = exp(-2t(a + 7)) > exp(-2t(a + 2/3 + 7)) = a^'^a^'^. 

Note also that the K2P model consists of all rate matrices where a = 7, which implies that 
= a^''^, and the JC models consists of all rate matrices where a = /3 = 7, which implies 

, A\B A\B A\B r-i 

that = Qq = arp . □ 

Proposition 12. Let Ti = Ti2j34, T2 = T13124, and T3 = Ti4[23. Then under the JC and K2P 
models, the polynomial 

Kq) = Qgggg — Qggtt 

satisfies the following properties: 
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1. l{q) = for all q G Mt^, 

2. l{q) > for all q G Mt,, ^ = 2, 3, 

3. l{q) > for all q E M^^, i = 2,3, and 

4- if q & J^Ti, for i = 2 or 3, and l{q) = 0, then the branch length of the internal edge is 
zero. 

Proof. To evaluate the polynomial l{q), we substitute for q the parametric expressions given 
in equation ([2]). Denoting parameters for trivial splits {i}|({l, 2, 3, 4} \ {i}) by a^, for 
q E Aixi we have 

U \ 1 2 3 4 12|34 1 2 3 4 12|34 

KQ) = Qgggg - Qggtt = aaaaacaaa^ - aaacarparpa^ . 

Since a^'^ = a^'^ in the JC and K2P models, the first claim follows. 
If g G A^T2; to establish the remaining claims note 

U \ 1 2 3 4 13|24 1 2 3 4 13|24 

= Qgggg - Qggtt = aQaQaQa^aj^ - a(.aQaj.arpac . 
Since a^^ = a^' ^for the JC and K2P models, and a^^'^^ = 1, this expression factors as 

U \ 1 2 3 4 /1 13|24n 

By Proposition [TTl all af^^ G (0, 1] , so l{q) > 0. Moreover, if all branch lengths are strictly 
positive, so is l{q). On the other hand, the only way this expression can equal zero with 
q G is if c^c '^"^ — 1- B^^^ then Proposition [H] implies the length of the internal branch 

is zero. 

Similar arguments show the claims for T3. □ 

Proof of Theorem^ Let K be any four element subset of the taxa. If Aij- fl Ai^ ^ 0, then 
when we marginalize to mixture models on the leaf set K the corresponding intersection 
is also non-empty. Since the claims of the theorem concern quartets, it suffices to restrict 
attention to the case of n = 4 taxa. 
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First suppose that the tree Ti is fully-resolved. By symmetry we may assume it is T12134. 
By Proposition [12], l{q) = if g G Mt^^^,^^, while /(g) > if g G M^^,^^^^ or M^^^^^^. By 
the linearity of /, this implies /(g) = if g G A^5, while /(g) > for g G provided T 

contains at least one of the resolved trees T13124 or T14123. This implies that if g G A^^flA^^^, 
then no quartet incompatible with tree Ti can appear among the trees of T. 

If Ti is the star tree, then from each of its three resolutions we obtain inequalities anal- 
ogous to those for /(g). These imply that T can only contain star trees. 

Finally, in the case that all T G T are binary and Ti ^ T, if g G Ais fl Ai^ then by 
replacing T by a subset T' we have g G Ais fl Ai^t ■ From the argument above it follows 
that for all T E T' and all quartets K, T\x = Ti\k- Thus we obtain the contradiction that 
T = Ti, and conclude no such g exists. □ 

Proof of Theoreml^ We first consider the case that T is a 3-leaf tripod tree, and Ti and T2 
are two of its contractions where one leaf has become an internal vertex. 

The model on a 3-leaf tree under the JC model has precisely 3 nontrivial Fourier pa- 
rameters, one per edge. We set the parametrization of that model, with edge parameters 
a,b,c G (0, 1], equal to the one for the mixture on Ti and T2, with edge parameters d,e 
and /, g respectively, and mixing parameter vr. This gives us, for fixed a, b, c, the following 
system of 4 equations in 5 unknowns: 



ab 



{l-iT)d + nf, 



ac = 



(1 — '7i)de + ng, 



be 



(1 - 7i)e + 71 fg, 



abc 



(1 -7i)de + 7ifg. 



It is not difficult to see that the values 



d = 0, e = be, f = b, g = c, tt = a 



(3) 
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give a solution to this system of equations. For the open models, however, we seek solutions 
where d,e,f,g,7r G (0,1) for fixed a,b,c G (0,1). A computation of the Jacobian of the 
system of equations at the values in equations ([3]) allows us to apply the implicit function 
theorem, and treat d as an independent variable in a neighborhood of the above solution. 
Hence, if we perturb d to d', with < rf' ^ 1, we obtain parameters in (0, 1) solving the 
system of equations. This shows that there is complete mimicking for the open models in 
the 3-leaf case. 

Finally we apply Proposition [5l Since any trees of the type specified in the statement 
of the theorem can be obtained by attaching fusion ends to the tripod tree and its two 
degenerations, we deduce the general result. □ 
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